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ABSTRACT 



This study applies multidimensional scaling (MDS) to a 
knowledge organization problem involving 12 concepts in pulmonary physiology. 
The concept structures were derived from individual medical and veterinary 
students before and after focused instruction. One hundred seventy medical 
students and 85 veterinary medical students completed a questionnaire about 
the similarity or relatedness of the concepts in pairs before and after 
instruction. Results from the study permit two general conclusions. The first 
is that the data pass the coherence test in that the pairwise similarity 
judgment data about the concepts met the scientific standards for goodness of 
fit. The concept organization data analyzed by MDS and concept educational 
achievement data contained in the final examination scores were trustworthy. 
However, the other conclusion is that the correspondence of the data is not 
established. Data suggest that knowledge assessment using structural methods 
like MDS produces results that are different from results obtained by 
knowledge assessment using multiple-choice tests. Each of these two 
approaches to student assessment in pulmonary physiology yields solid data, 
but for practical purposes, the data have no common variance. (Contains 2 
tables and 24 references.) (SLD) 
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Multidimensional Scaling Assessment of Medical and Veterinary Student 
Knowledge Organization of Pulmonary Physiology Concepts 



Research in a variety of fields including physiology (1-3), medicine (4), research 
methods and statistics (5), computer programming (6), psychology (7), biology (8), and 
teacher education (9) has tried to capture or represent the cognitive structure, the 
intellectual organization, of concepts acquired by learners at different educational levels. 
This research is based on two fundamental assumptions. First, the ability to organize, 
simplify, and store a large body of educational material in a specific field is essential for 
the development of expertise (10). The coherence of information increases as individuals 
learn to organize and simplify data and concepts for storage in memory. Meaningful 
learning involves the ability to structure educational material, not just the ability to 
acquire material in volume. Second, the organization of concepts in memory mediates 
their recall and use. As the conceptual organization of educational material is simplified 
its retrieval from memory becomes automatic and the cognitive workload needed for 
recall is reduced. Thus as pointed out by Nobel Laureate H.A. Simon in his landmark 
book. Models of Thought (11), the ways in which experts and learners in a field of study 
intellectually organize its key concepts affects their ability to retrieve and use the 
material. 

Studies in medical education have been at the leading edge of research on 
knowledge organization. This work has been done using several different research 
methods including qualitative concept mapping (1, 12-14), use of “thinking aloud” 
protocols in the cognitive science tradition (15, 16), multidimensional scaling (MDS) (4), 
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assessment of semantic structures (17, 18), and concept network analysis using the 
Pathfinder scaling algorithm (2, 3). While the individual studies within this body of 
research vary in scope, measurement methods, data analysis procedures, and scientific 
rigor they all address the research goal of organizing medical information with coherence 
and parsimony. 

Quantitative analysis of concept organization via MDS or Pathfinder network 
scaling holds particular promise for medical education research. In contrast with 
qualitative approaches to research on concept formation and organization, these 
quantitative methods are rigorous, can be replicated, and allow for statistical analyses of 
concept data sets. For example, such analyses permit investigators to systematically 
study issues including expert-novice differences in concept organization (6, 7), changes 
in concept structure due to educational interventions (5), correlations between indexes of 
concept organization and independent measures of educational achievement (3, 5, 7), and 
variation among experts in concept organization (2). 

MDS is one means to display the judged similarity or difference of a set of stimuli 
(e.g., pulmonary physiology concepts) on a spatial map (19). Pairs of stimuli (concepts) 
are judged in terms of their similarity or difference. Following quantitative analysis, 
stimuli that are consistently judged to be similar are graphically displayed in geometric 
space to be closer than stimuli that are consistently judged to be different. 

There is controversy about the propriety of MDS to map judgments about 
nonphysical entities like concepts in the medical basic sciences. Some investigators 
question the geometric assumptions that underlie such analyses and express concern 
about the interpretability of results when the entities being studied are intangible (20). 
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By contrast, other researchers have shown in theory (21) and in practice that MDS is a 
useful method to analyze judgments about such abstract entities as physics principles (22) 
and concepts in developmental psychology (23). 

This study applies MDS to a knowledge organization problem involving 12 
concepts in pulmonary physiology. The concepts are chemoreceptors, lung gas 
exchange, ventilation, spinal cord, perfusion, intrapleural pressure, respiratory mechanics, 
surface tension, resistance, control of breathing, diffusion, and partial pressure. Concept 
structures were derived in several dimensions from individual medical and veterinary 
students before and after a period of focused instruction. The study addresses three 
questions. First, does the research task we devised about pulmonary physiology concept 
formation and representation yield data that are suitable for MDS analysis? Second, can 
an MDS solution account for a meaningful proportion of variance in medical and 
veterinary students’ concept representations? Third, do individual differences in the way 
in which medical and veterinary students intellectually organize the pulmonary 
physiology concepts, captured by MDS, correlate with course examination achievement? 

Methods 

Participants. One hundred seventy Northwestern University medical students and 
85 University of Wisconsin veterinary medical students volunteered to participate in the 
study during the Fall of 1996. They supplied preinstruction and, with some attrition, 
postinstruction data. This occurred in the context of a first year physiology course 
section or a complete course at each institution, respectively. 

Measurements. A questionnaire was constructed that presented all possible 
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n(n - l)/2 = 66 pairs of the 12 concepts randomized for presentation order (left-right) 
and sequence (1-66). Instructions directed research participants to provide a judgment 
about the degree of similarity between or relatedness of the two concepts in each pair on 
a scale of 0 (completely unrelated) to 9 (highly related). This judgment task was 
completed on both occasions by most of the medical and veterinary medical students in 
approximately 20 minutes. 

Data Collection. Northwestern University medical students and University of 
Wisconsin veterinary medical students completed the questionnaire in class, immediately 
before and after in instructional unit on pulmonary physiology that lasted three weeks. 

Data Analysis. Data analyses were performed using the individual differences 
(INDSCAL) option of the ALSCAL algorithm in SPSS. For each institution 
(Northwestern, Wisconsin) and for each occasion (preinstruction, postinstruction) the 
MDS analysis yielded solutions in dimensions ranging from two through six. For each 
dimensional solution, coefficients of S-Stress and Stress were calculated as indexes of 
“goodness of fit” of the data to the statistical algorithm. In addition, an index of variance 
explained or accounted for (R ) was calculated for each dimensional solution. This is a 
measure of internal consistency for each group of students on each occasion. An increase 
in MDS R from preinstruction to postinstruction about a set of concepts for a student 
group indicates the concepts are becoming increasingly coherent, conceptually 
meaningful, for the students. Finally, individual students’ subject weights for the MDS 
dimensions contained in the most interpretable solution were correlated with their scores 
on a course examination covering the pulmonary physiology material. 
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Results 



A summary of the data analysis is given in Table 1 . MDS solutions ranging from 
two to six dimensions are shown for the Northwestern University medical students pre 
and post instruction data and for the University of Wisconsin veterinary medical students 
pre and postinstruction data. 

Table 1 here 

Attrition from the study occurred for the volunteer students in both groups. For 
the Northwestern medical students the pretest to posttest loss was from 170 to 131 
students, a drop of 23%. For the University of Wisconsin veterinary medical students the 
pretest to posttest loss was from 85 to 73, a 14% reduction. Students lost to followup 
were not present or refused to participate on the occasion of the posttest, or refused to 
participate when contacted subsequently. 

In response to the first study question, the research task clearly produced data 
suitable for MDS analysis. Of the various dimensional solutions displayed for the two 
institutions (Northwestern, Wisconsin) and the two occasions (pre and postinstruction), 
the four dimensional solution produced the most interpretable results. The rule-of-thumb 
in INDSCAL research is that a dimensional solution should have a Stress level at or 
below .15 to provide a good fit of the data to the computing algorithm (19). In addition, 
in studies like this investigation it is important to demonstrate a substantial increase in 
variance accounted for (R 2 ) from pretest through educational intervention to posttest as 
an index of increased conceptual coherence among the medical and veterinary medical 
students. 
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The four-dimensional MDS results displayed in Table 1 clearly achieved the 
minimum expectations for “goodness of fit” (Stress) and variance accounted for (R 2 ). 

For the Northwestern data the Stress levels at pre and posttest are .162 and .150, 
respectively. The R 2 from pretest to posttest increases from .552 to .645, a 17% boost. 
For the University of Wisconsin data the Stress levels at pre and posttest are .169 and 
.148, respectively. The R from pretest to posttest jumps from .500 to .673, an increase 
of 35%. Thus on grounds of “goodness of fit” (reduced Stress) and improvement in 
conceptual coherence (increased R ) the four-dimensional MDS solution provides a 
parsimonious organization of the pulmonary concept judgment data. 

Table 2 presents correlations of individual students’ subject weights for each of 
the four dimensions (pre and postinstruction) with their scores on course examinations 
created independently at Northwestern University and the University of Wisconsin. The 
reliability coefficient (KR-20) for the Northwestern examination was .79. For the 
University of Wisconsin examination the reliability coefficient was .86. 

Table 2 here 

Outcomes from this analysis show that, with one exception, there is no correlation 
between students’ structural knowledge organization of the 12 pulmonary physiology 
concepts represented by MDS and their achievement on course examinations covering 
the same material. The sole exception is a statistically significant negative correlation 
between students’ scores on the third MDS dimension and final examination scores for 
the University of Wisconsin veterinary medicine pretest data. From an array of 16 
correlation coefficients, a single significant outcome may also be an artifact of chance. 
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Discussion 



Results from this study permit two general conclusions. The first conclusion 
concerns what psychologist Kenneth Hammond (24) would call the coherence or internal 
consistency of our data. The second conclusion addresses Hammond’s notion about the 
correspondence of data, their link with different measures of a presumably similar 
construct. 

Our data clearly pass the coherence test. The pairwise similarity judgment data 
about the pulmonary physiology concepts, arrayed in four dimensions by the INDSCAL 
MDS algorithm, meet scientific standards for “goodness of fit” (Stress) and variance 
accounted for (R 2 ). Students’ final examination scores at both Northwestern and 
Wisconsin also reach acceptable levels of test reliability. Thus customary psychometric 
indexes of coherence and internal consistency show that the (a) concept organization data 
analyzed by MDS and (b) concept educational achievement data contained in final 
examination scores are trustworthy. 

However, the correspondence of our data is not established. The data suggest that 
knowledge assessment using structural methods like MDS produces results that are 
different from results obtained by knowledge assessment using multiple-choice tests. In 
this study, each approach to student assessment in pulmonary physiology yields very 
solid data, but for practical purposes the data have no common variance. 

This study outcome is in sharp contrast with past research where correlations have 
been demonstrated between measures of student structural knowledge and customary 
measures of school achievement (3, 5, 7, 9). However, the studies just cited used the 
Pathfinder scaling algorithm to assess knowledge organization rather than MDS. This 




suggests that the approach used to assess knowledge organization among concepts (or 
other entities) may govern an investigator’s ability to link student structural knowledge 
with acquired fund of knowledge. 

Future work by our research group will continue to assess medical and veterinary 
student knowledge organization in both basic science and clinical domains. An essential 
feature of this work will be a search for ways to establish the correspondence of 
measures of structural knowledge with other measures of knowledge acquisition and use. 
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